The need for moral competency in autonomous agent architectures

Artificial intelligence and robotics are rapidly advancing in their quest to build truly autonomous agents. In particular, autonomous robots are envisioned to be deployed into our society in the not-so-distant future in many different application domains, ranging from assistive robots for health-care settings, to combat robots on the battlefield. Critically, all these robots will have to have the capability to make decisions on their own to varying degrees, as implied by the attribute “autonomous”. While these decisions might often be in line with what the robots’ designers intended, I take it to be self-evident that there can, and likely will be cases where robots will make inadequate decisions. This is because the world is “open”, with new entities and events appearing that could not have been anticipated by robot designers (e.g., Talamadupula et al (2010)). And even if the designers’ response to the world’s openness was to endow their robots with the ability to adapt to new situations and acquire new knowledge during their operation, so much for the worse, because learning capabilities in autonomous robots leave even less control in the hands of the designers and thus open up the possibility for inadequate decisions. Note that “inadequate” covers a wide spectrum of decisions, from the simplest cases of being “sub-optimal”, to the most serious cases of deep moral and ethical violations. It is not necessary to conjure up a Terminator-like scenario where a self-righteous AI system decides that humans are a nuisance and need to be eradicated; simple social robots causing harm to their owners because of their lack of emotional awareness and availability will do Scheutz (2012). In this chapter, I will make the plea for developing moral capabilities deeply integrated into the control architectures of such autonomous agents, following previous such appeals (e.g., Wallach and Allen (2009); Arkin and Ulam (2009)), albeit for different reasons. For I shall argue that any ordinary decision-making situation from